Repository navigation
Conversation
Adds docs/ADDING-A-LANGUAGE.md walking through every step a contributor
needs to add a new language extractor:
1. Source a tree-sitter wasm grammar — covers the three real-world
paths (already in tree-sitter-wasms, pre-built release artifact,
build from source via tree-sitter-cli's bundled wasi-sdk).
2. Probe the AST with a small scratch script before writing code.
3. Register in src/types.ts + src/extraction/grammars.ts.
4. Type-check before adding extraction logic.
5. Pick a pattern: LanguageExtractor config (procedural / OO) or a
self-contained extractor class (declarative / template / non-OO).
6. Map onto existing NodeKind / EdgeKind values.
7. Tests + end-to-end CLI smoke.
8. PR description checklist.
Each section points at the existing extractors as worked examples
(R for the OO path, HCL/SQL/Liquid for the custom path, Pascal+DFM
for the cross-format case). README.md and CLAUDE.md gain a one-line
pointer to the cookbook.
Closes colbymchenry#55.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sections 3, 5a, 5b previously taught the monolithic-file pattern that PR colbymchenry#116 obsoletes. After colbymchenry#116, adding a language is one new file in src/extraction/languages/ + 2 lines in registry.ts (and an optional 1-line addition to the Language union for TypeScript narrowing). Updated: - Section 3: full rewrite. Was 3-file mutation (types.ts, grammars.ts, CLAUDE.md). Now: 1 LanguageDef file + registry import + Language union entry. Includes a "why per-file" sidebar pointing at the cross-PR conflict bottleneck the registry resolves. - Section 5a: drops the EXTRACTORS-map registration step. The extractor is referenced from the LanguageDef directly. - Section 5b: drops the tree-sitter.ts dispatch wiring. customExtractor on the LanguageDef takes the dispatch — no per-language if-branches. Section 1 (sourcing wasm), Section 2 (probing AST), Sections 6/7/8 (NodeKind mapping, tests, PR checklist), and the existing-extractors reference table are unchanged — those parts of the workflow didn't change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add CONTRIBUTING.md covering development setup, project architecture, testing conventions, commit message format, and PR workflow. References docs/ADDING-A-LANGUAGE.md for language addition contributions.
ymstar
pushed a commit
that referenced
this pull request
Oct 22, 2026
…ymchenry#720 (colbymchenry#748) The per-word path fix (colbymchenry#745) brought the backend to parity but not above: the project name still gave the lexically-matching stack a residual dir match + an FTS class-name match, so a backend query that included the project name still ranked the frontend at/above the backend. Derive the project name from go.mod module / package.json name / repo dir, and treat a query word matching it as non-discriminative: drop it from path relevance and from codegraph_explore's PascalCase type-disambiguation bias (reporter's suggestions #1/#2) — unless it's the only query word, so a bare project-name search still scores. Narrow by construction: the down-weighting fires ONLY when a query word matches the derived project name (≥5 chars), so every query that doesn't name the project is byte-identical. On the reporter's repro the backend controllers now top a backend question that includes the project name; queries without it, bare project-name queries, and normal symbol queries are unchanged. Query-time only (no re-index). Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ymstar
pushed a commit
that referenced
this pull request
Oct 22, 2026
…C lua v0.4.1, tree-sitter-luau 1.2.0 pin, both default-routed (colbymchenry#1384) R7b batch 4 #2 (docs/design/lua-luau-kernel-port-checklist.md is the authoritative quirk list). ONE walker for both dialects (ccpp precedent) — the differences are exactly four: luau's type_definition aliases, the `export `-slice isExported hook, the return-type signature suffix, and the grammar handle. Grammar prep is kernel-side only, no wasm change: lua is the SECOND vendored-grammar-C language (the vendored wasm is the v0.4.1 tag, a revision not on crates.io — tag artifacts compiled via build.rs, shas pinned); luau is a plain crate pin =1.2.0 whose tarball is sha-identical to the tag (the swift tag≠crate divergence does not recur). Grammar-parity rows replace the bump gate entirely. Preserved bug-for-bug (all probe-pinned): the require/visitNode-hook ASYMMETRIES (top-level requires — including inside top-level if/for/while — mint import nodes while the identical body-level statement emits `calls "require"`; top-level `local x = foo()` initializers are invisible while global `x = foo()` calls emit), the BFS string-win inside require args (`require(script:WaitForChild("Kid"))` → import Kid) and Roblox instance paths, receiver-QN methods (`M.sub.deep::chained`, `_G::installed`, stack-QN nested globals like `render::leakedGlobal`), the raw-text callee world (colon forms with `self` never stripped, bracket callees, newline-glued chains byte-verbatim, the `(handler)` paren-conversion), LUA_SPEC function-as-value capture with the `M.cb = cb` param-storage skip and first-occurrence dedupe, LuaDoc `---` keeping a leading `- ` plus `--!strict` joining docstring chains (block-comment docstrings keep interior CRLF bytes), variable nodes at the IDENTIFIER with positional value pairing, duplicate same-(kind,name,line) ids, and the lua↔luau isExported wire divergence (lua functions: flag absent; luau functions: present-false; methods: absent in both; variables: present-false in both; `export type`: true). Gates: parity sweeps first-run 0-diff on kong/lazy.nvim/lua-resty-core (lua) + lune/Fusion (luau) — 1,734 clean files byte-parity, deferrals 1/0/0/3/8 matching the survey's both-arm predictions exactly (kong's 1 = a deliberately invalid fixture; luau's = grammar-inherent generic type packs and default type params); full-init dumps byte-identical kernel-vs-wasm ×4 (kong 157,650 dump lines); kernel-lua-parity suite (both torture fixtures + in-memory CRLF variants + glue-chain, duplicate-id, and cross-dialect defer pins + kernel-arm wire-flag pins); full suite 2,647 green ×2 with CODEGRAPH_KERNEL_EXPECT=1. DEFAULT_ROUTED += lua, luau (18 langs). Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
ymstar
pushed a commit
that referenced
this pull request
Oct 22, 2026
…ame-collision matches (CG-10, colbymchenry#1500) Explore's per-file relevance awarded +50/+10/+3/+1 by match class and admitted anything scoring >= 3. Neither half held up: the tier said HOW a symbol reached us, never whether the match was evidence, and an absolute floor admits noise on any repo where the top file scores 50+. Three scripts/agent-eval/*.mjs harnesses took 63% of this repo's own "how does explore allocate its output budget" answer on nothing but an unused `const explore` and a `const BUDGET`. Four levers: - KIND WEIGHT (RELEVANCE_KIND_WEIGHT): callables and types 1.0, members ~0.5, variable/constant/parameter 0.15-0.35. A weak-kind symbol with no usage edge anywhere in the graph (`contains` excluded — nesting is not usage) drops to 0.08. Only weak kinds in the top two tiers pay for the DB probe; the subgraph's own edges answer most cases free. No measurable latency change (210 vs 211 ms/call, n=12 interleaved). - PERIPHERAL CAP: nodes >=2 hops from any match accumulate into a bucket capped at 5. Uncapped they added a flat +1 each, so a file grew more relevant by being bigger — parse-session.mjs reached 22 off one constant plus twelve unrelated symbols. - RANK PENALTY: generated files x0.3, low-value x0.5, applied to the score AND the graph mass. Score alone would not have fixed colbymchenry#1500 — the generated CRUD carries MORE graph mass than the hand-written use-case, and graph mass outranks score in the comparator. Self-normalizing, never a hard exclusion. - RELATIVE FLOOR: clamp(topScore * 0.2, 1, 10). Capped at one full-strength direct match so concentration elsewhere can never exclude one (without it a named-seed-heavy file pushed the floor to 21 and dropped a file the agent had named by class name). Backfills to 3 candidates when it would leave fewer, and drops the evidence requirement rather than return nothing at all. excludeLowValueFiles was dead config — declared per tier, read nowhere; the test/spec exclusion has been unconditional for a while. Removed. The real gap was the detector: `isLowValue` anchored on a leading `/`, so a repo-ROOT `test/` dir (express, cobra, most of npm and Go) never matched — express's routing question spent 59% of its envelope on three test files. Anchored at `^` too, and the filter now runs before the floor and judges "are there other candidates?" on the whole gather. Measured before/after on the same indexes (baseline bd86ad2): - payroll-go fixture: generated 57.4% -> 23.5%; answer 25.6% -> 61.5%; cycle.go delivered 0 -> 38.9%. Generated ranks colbymchenry#3/colbymchenry#4, was #1/#2. - self-query fixture: eval scripts 72% -> 0%; tools.ts ranks #1. - express "route a request": 59% to test/* -> lib/application.js + lib/response.js - cobra x3, codegraph "indexing pipeline": byte-identical (control) Diagnostic gains a per-file penalty multiplier and NodeKind mix, so "why did this file score X" is legible. Selection stages reordered to match the pipeline. CG-6's gates flip from it.fails to live regressions except the byte-split ones, which stay open for CG-12 (allocation still follows file size within the ranked set). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ymstar
pushed a commit
that referenced
this pull request
Oct 22, 2026
The agent's next action after a codegraph_explore is free ground truth about
whether the response was enough, and the harness was discarding it. Every run
now bucketed: explored again (insufficient), Read a file we returned
(allocation -- right file, wrong bytes), Read a file we did not return (recall),
Grep/Glob (recall, weak), or moved on (sufficient). The buckets are chosen so
each one names a distinct fix.
The classifier lives in parse-run.mjs next to the occupancy math and takes raw
events, so parse-session.mjs reuses it for interactive runs -- no new
scripts/agent-eval/*.mjs, which would score into the self-query eval fixture's
corpus.
Four rules, three of them found by validating against real transcripts rather
than reasoned up front:
* A call in the SAME assistant message as the explore predates its response,
so it is not a verdict on it. Stepped over, counted as `concurrent`.
* ToolSearch/TodoWrite carry no signal; the call behind them is the verdict.
* SUBAGENTS ARE A SEPARATE THREAD. Claude Code interleaves a subagent's calls
into the same stream under parent_tool_use_id -- verified on a live
excalidraw run where a delegated search's greps landed between the parent's
own calls. Matching reactions across threads scored the subagent's grep as
the parent's verdict on an explore it never saw. A delegation is judged by
what the subagent did FIRST: before that, the same run reported 33%
sufficient while the subagent was off grepping for the file, which is the
one direction of error a tuning metric must not have. In interactive
sessions the subagent is a separate FILE instead, so parse-session.mjs
stitches the threads back with the toolUseId in agent-*.meta.json.
* A re-read of a file an EARLIER explore shipped is still allocation, not
recall -- filing it as recall aims the fix at the wrong end of the pipeline.
Shell file access counts too (`sed -n 100,200p f` reads, `grep`/`find` search),
since both arms have Bash and counting only the Read tool would score those
explores as sufficient. A heredoc or redirect is writing, not reading.
Validated by hand on cg22/ab-express/run-baseline-1 (explore, explore, Read of
lib/response.js which explore #2 returned -- the colbymchenry#1500 allocation bug as a
bucket instead of a hunch; the new-build arm is 100% sufficient) and on
cg15/ab-express/run-new-2 (four explores, the last returning lib/utils.js which
the agent then read at offset 195). Swept over all 76 A/B logs on this machine:
0 crashes, 176 calls bucketed. --selftest covers every bucket, both thread
rules, delegation, shell reads/searches and errored calls: 46/46.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add
CONTRIBUTING.md— a comprehensive guide for contributors covering development setup, project architecture, testing conventions, commit message format, and PR workflow.This branch is built on top of @andreinknv's work from PR #97 (
docs/ADDING-A-LANGUAGE.md), whose commits are preserved with original authorship. The contributing guide references their cookbook in the Adding a New Language section.Changes
CONTRIBUTING.mdWhat it covers
docs/ADDING-A-LANGUAGE.md)Attribution
Based on @andreinknv's ADDING-A-LANGUAGE.md cookbook (cherry-picked with original commit authorship preserved). Used with consideration — the upstream PR was closed without merging, so the cookbook content was incorporated by cherry-picking the original commits to retain proper git attribution.